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The quantum anomalous Hall (QAH) state is a two-dimensional bulk insulator with a non- 
zero Chern number in absence of external magnetic fields. Protected gapless chiral edge 

^ • states enable dissipationless current transport in electronic devices. Doping topological in- 

<U ■ 

sulators with random magnetic impurities could realize the QAH state, but magnetic order 
is difficult to establish experimentally in the bulk insulating limit. Here we predict that the 

s 

single quintuple layer of GdBiTe 3 film could be a stoichiometric QAH insulator based on ab- 

o 

initio calculations, which explicitly demonstrate ferromagnetic order and chiral edge states 
inside the bulk gap. We further investigate the topological quantum phase transition by tun- 

> 

IT") . ing the lattice constant and interactions. A simple low-energy effective model is presented to 

oo 
oo 

o 



capture the salient physical feature of this topological material. 



Recently time-reversal-invariant topological insulators(TIs) have attracted broad attention in 
fields of the condensed matter physics, material science and electrical engineering 1 4 . Quantum 
spin Hall(QSH) insulators^ are two dimensions(2D) TIs with a bulk energy gap and gapless he- 
lical edge states which are protected by the Z 2 topological invariant 7 . Both 2D and 3D TIs are 
natural platforms to realize the QAH when the time-reversal symmetry is spontaneously broken 
by the ferromagnetic order^-R The QAH state is the quantum version of the intrinsic anomalous 
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Hall effect , and is characterized by a bulk 2D Chern number and gapless chiral edge states with- 
out the external magnetic fiel d 8 * 13 ^ , and is sometimes also referred to as the 2D Chern insulator. 
The QSH state has two counter-propagating edge states; doping the 2D TIs with magnetic impu- 
rities could break the time reversal symmetry, annihilate one edge state and leave the other intact, 
thus realizing the QAH state with chiral edge state. Alternatively, doping the surface of 3D TIs 
with magnetic impurities breaks time reversal symmetry, and opens up a gap for the Dirac surface 
state^LULL^EE'. Chiral edge states exist on the magnetic domain walls on the surface^. In the QAH 
state, electrons move like cars on a highway, where oppositely moving traffics are spatially sepa- 
rated into opposite lanes 1 . Realizing such a dissipationless transport mechanism without extreme 
condition could greatly improve the performance of electronic devices. 

Normally, ferromagnetic order among magnetic impurities in randomly doped semiconduc- 
tors are mediated by the free carriers ^ it is difficult to establish magnetic order in the bulk insu- 
lating limit. For this reason, we search for stoichiometric ferromagnetic insulators with a nonzero 
Chern number for the 2D band structure. Rare earth elements have partially filled f electrons, in 
particular, Gd 3+ has exact half filled f electrons. It is possible to realize a stoichiometric QAH 
insulator by replacing Bi by Gd in the Bi 2 Te3 family of Tld^EQ. GdBiTe3 has a stable structure 
phase, synthesized thirty years agcPl In this work, we predict that stoichiometric QAH insulators 
could be realized in the single quintuple layer(SQL) of GdBiTe 3 , based on ab-initio calculations. 

Crystal structure and band structure 

GdBiTe 3 has a rhombohedral crystal structure with the space group R3m. Its crystal consists 
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of close-packed layers stacked along [111] direction with the A-B-C- • • order. The quintuple layer 
(QL) (Te-X-Te-X-Te) structure is the basic crystal unit, where X presents Bi and Gd with a certain 
pattern, similar to the case of LaBiTe^H It has strong coupling within one QL and weak van der 
Waals coupling between neighbor QLs, so this material can be grown as two-dimensional (2D) 
thin film along [111] direction. In addition, though its lattice type and lattice constant have been 
measured experimentally 22 , positions of Bi and Gd have not been clearly resolved yet. Since Bi 
and Gd have different electronegativity, it is possible to grow the non-mixing structure as Te-Gd- 
Te-Bi-Te with Molecular beam epitaxy (MBE) technique. This crystal structure of GdBiTe3 is 
shown in Fig. la. It holds the three-fold rotation symmetry C 3 with z axis as the trigonal axis 
and the reflection symmetry with x axis as its normal axis. Compared with Bi 2 Te3 18 , the inversion 
symmetry is broken for this structure. Since SQL GdBiTe3 film is the simplest system to study 
the QAH effect, in this work we focus on its SQL structure with Tel-Bi-Te2-Gd-Tel', where Tel 
and Tel' are marked for the Te closest to Bi and Gd layer separately, and Te2 denotes the Te in the 
center of this SQL. 

All ab-initio calculations are carried out in the framework of density functional theory (DFT) 
with Perdew-Burke-Ernzerbof-type generalized-gradient approximation 23 . Both BSTATE package 24 
with plane-wave pseudo-potential method and the Vienna ab initio simulation package (VASP) 
with the projected augmented wave method are employed. The k-point grid is taken as 12 x 12 x 1, 
and the kinetic energy cutoff is fixed to 340eV in all self-consistent calculations. A free standing 
slab model is employed with SQL. Its lattice constant (a = 4.16A) is taken from experiments^, 
and the inner atomic positions are obtained through the ionic relaxation with the force cutoff 
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O.OOleV/A. The spin-orbit coupling (SOC) is taken into account, because of its importance to 
realize the QAH effect. GGA+U method 25 is also employed to study the strong correlation effect 
in GdBiTe 3 because of the existence of narrow occupied f bands of Gd. We find that the ferromag- 
netic phase is more stable than the non-magnetic and collinear anti-ferromagnetic phases for SQL 
GdBiTe 3 . In addition, the ferromagnetic phase with magnetic moment along [111] direction has 
lower energy than that with the magnetic moment along [010] and [110] directions. All GdBiTe3 
calculations are carried out here with the ferromagnetic moments along [111] direction. 

The ferromagnetic phase of SGL GdBiTe 3 is an insulator state with exact magnetic moment 
S = 7/2. Because of the lattice similarity between GdBiTe3 and GdN(or EuO) with [111] direc- 
tion, the known magnetic exchange mechanisms in GdN or EuO could provide possible explana- 
tion for the ferromagnetic state in SQL GdBiTe 3 . The first mechanism is a third-order perturbation 
process^. A virtual excitation, which takes a 4f to a 5d state, leads to a f-f interaction through the 
d-f exchange due to the wave-function overlap between neighboring rare-earth atoms. Recently, 
Mitra and Lambrecht 27 , based on ab-initio calculations, presented another magnetic mechanism 
for the ferromagnetic ground state in GdN. The anti-ferromagnetic ordering, between N p and Gd 
d small magnetic moments, stabilizes the ferromagnetic structure between nearest neighbor Gd 
atoms due to the d-f exchange interaction. In addition, our calculations also indicate the similar 
anti-ferromagnetic ordering between Te p and Gd d small magnetic moments. 

Bi 2 Te 3 , LaBiTe 3 and GdBiTe 3 have very similar SQL structure. First of all, Bi 2 Te 3 have both 
the inversion symmetry and the time-reversal symmetry. However, inversion symmetry is broken 



4 



in LaBiTe 3 , and both inversion symmetry and time-reversal symmetry are broken in GdBiTe 3 . 
The band evolution from Bi 2 Te 3 to LaBiTe 3 , and finally to GdBiTe 3 is shown in Fig. lc-e. The 
bands of Bi 2 Te 3 have double degeneracy because of both inversion and time-reversal symmetries. 
Its energy gap is calculated to be about 0.19eV, consistent with the experiments 28 . The bottom 
of the conduction bands at T point with a "V" shape originates from the p xy orbitals of Bi, Tel 
and Tel'. The top of the valence band at the Y point, which lies below valence bands at other 
momenta, originates from the p XjV orbitals of Te2. For LaBiTe 3 , the double degeneracy of the 
bands is lifted except at the time-reversal points, due to the lack of the inversion symmetry. Though 
the bottom of conduction bands still shows the "V" shape, the top of the valence bands moves to 
T, which mainly originates from the p x>v orbitals of Te2 and Tel'. The schematic evolution of the 
top of valence bands and the bottom of conduction bands from LaBiTe 3 to GdBiTe 3 is shown in 
Fig. 2a-b. The Kramers double degeneracy at the T point in GdBiTe 3 is broken due to the lack 
of time-reversal symmetry. The band inversion occurs between the \p x + ip y , Bi + Tel, 1} and 
\Px + ipy-* Tel' + Te2, f) due to the strong SOC and the large magnetic moment. The existence of 
this band inversion is the key point to realize the QAH effect in this SQL GdBiTe 3 system. 

Fig. 3a shows that the energy gap is quite sensitive with the lattice constant. Smaller lattice 
constant a increases the band inversion. Conversely, increasing the lattice constant a would de- 
stroy the band inversion. The QAH effect exists for a < 1.008a , where a is the experimentally 
determined bulk lattice constant. Because the SQL GdBiTe 3 is very thin, its lattice constant could 
be controlled by the substrate, giving us a broad tunability range. In addition, due to the narrow f 
bands, the strong correlation needs to be checked by GGA+U calculations. First of all, the band 
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structure calculations show that the occupied f bands are located at — QeV below Fermi level (FL), 
and the unoccupied f bands are located at 2.5eV above FL. Since both occupied and unoccupied f 
bands are quite far from FL, correlation effects should not influence bands very close to the Fermi 
level. Our calculations indicate that this conclusion is true. The dependence of the energy gap on U 
(from OeV to 8eV) is shown in Fig. 3b. The correlation U lifts up the top spin-down valence band, 
and pulls down the bottom spin-up valence band. Therefore, the strong correlation U likes to pull 
GdBiTe 3 from the topologically non-trivial QAH phase to the topologically trivial ferromagnetic 
insulator phase. But the energy gap is much more sensitive with the lattice constant than with the 
correlation U. We have shown that the SQL GdBiTe 3 is close to the quantum critical point between 
the QAH state and topologically trivial magnetic insulator state, and it is possible to realize the 
QAH state by tuning the lattice constant with proper substrates. Also the 2D Dirac-type band dis- 
persion shows up at the critical point, shown in Fig. 3c. This is very similar to the case of QSH 
state at the critical poinP. 

Topological chiral edge states 

The existence of topologically protected chiral edge states is the direct and intrinsic evidence 
of the QAH phase. It is important to show the explicit features of these topologically protected chi- 
ral edge states. In order to calculate the edge states, we employ the tight-binding method based on 
maximally localized Wannier functions (MLWF), developed by Vanderbilt and his co-workers^, 
in the framework of ab-initio calculations. The edge with Tel' and Bi terminated along the [11] 
direction is chosen to show the edge states, shown in the inset of Fig. 4b. Because the topological 
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nature is completely determined by its bulk electronic structure, here we ignore the edge recon- 
struction of the atoms on the edge, and also make another approximation that the MLWF hopping 
parameters close to the edge are the same to ones of the bulk. The bulk MLWF hopping param- 
eters are obtained from the ab-initio calculations of the SQL free-standing GdBiTe 3 slab. Similar 
to the method described in our previous workP', we use an iterative method 30 to obtain the edge 
Green's function of the semi-infinite system with an edge along [11] direction. The local density 
of states (LDOS) can be calculated with the imaginary part of these edge Green's functions, shown 
in Fig. 4a,b. The LDOS in Fig. 4a shows two edge states for topologically non-trivial GdBiTe 3 
with the experimental lattice constant a . One topologically trivial edge state only stays within 
the valence bands, while the topologically non-trivial edge state ties the conduction band with 
the valence band. This topologically non-trivial edge state is chiral, and its Fermi velocity in the 
energy gap is vp ~ 3.1 x 10 5 m/s (or 3.86el/ • Bohr). The edge states of topologically trivial 
ferromagnetic GdBiTe 3 with the lattice constant a = 1.0 2a are shown in Fig. 4b. Both of these 
two topologically trivial edge states only stay within the valence bands, and do not connect the 
valence and conduction bands. 

Low-energy effective model 

As the topological nature is determined by the physics near the T point for this material, 
it is possible to write down a four-band effective Hamiltonian to characterize the low-energy 
long-wavelength properties of the system. Starting from the four low-lying states \B_, — ~ f) = 
ci\p x - ip y ,Tel,t) + c 2 \p x - ip y ,Bi,t), \B + , \ l) = d\p x + ip y ,Tel,l) + c 2 \p x + ip y ,Bi,l), 



7 



|T+,§ t) = d 1 \ Px + i Py ,Te2,t) + d 2 \ Px + i Py ,Tel',t) and |T_, -§ |) = di|p x - Te2, | 



) + d 2 \p x + ip y , Tel', t) at the T point, where c\, c 2 , di and c/ 2 are constants and ±|(§) are the an- 
gular momenta in z direction. In addition, the "±" subscripts are used to present the z component 
of orbital angular momentum l z = ±1. Such a Hamiltonian can be constructed by the theory of 
invariants for the finite wave vector k. On the basis of the symmetries of the system, the generic 
form of the 4x4 effective Hamiltonian can be constructed up to the order of 0(k 2 ), and the pa- 
rameters of the Hamiltonian can be obtained by fitting to ab initio calculations. First we consider 
the Te-La-Te-Bi-Te system without magnetic order. The important symmetries of this system are 
(1) time-reversal symmetry T, (2) reflection symmetry o x with x axis as the normal axis, and (3) 
three-fold rotation symmetry C3 along the z axis. In the basis of \B + , ~ \), \T + , | f), |£>_, — | t) 
and |T_, — I |), the representation of the symmetry operations is given by T = K, ■ ia y <g> r 2 , 
a x = —ia x eg) I and C3 = exp(— i^rr J z ), where K, is the complex conjugation operator, a x ' y ' z and 
T x, y ,z d enote p au ii matrices in the spin and orbital space, respectively and J z is the angular 
momentum in z direction. For low energy effective model, we ignore the in-plane anisotropy and 
impose continuous rotational symmetry R{6) = exp(—i6J z ). By requiring these three symmetries, 
we obtain the following generic form of the effective Hamiltonian: 

^ M(k) Ak + -iA e k_ A'k 2 _ ^ 
Ak^ -M(k) A'k 2 _ 
tA e k + A'k 2 + M(k) Ak_ 
A'k% Ak + -M(k) 
with k± = k x ± ik y , e (k) = C + D(k 2 x + k 2 ), M(k) = M + B{k 2 x + k 2 y ). The block diagonal 
part is just the BHZ model 5 , and the off-block diagonal ±iA e k±, Nk\ terms breaks the inversion 



\ 



*o(k) 



(1) 
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symmetry. Next we add the ferromagnetic exchange term which breaks time reversal symmetry, 



Hp 



-g B M 



\ 



g T M 



g B M 



\ 



-g T M 



(2) 



with g B and g T as Lande-g factors for two different orbitals and M the exchange field introduced 
by the FM ordering of Gd with half-occupied f electrons. Then the full Hamiltonian can be written 
as H = Ho + Hex- By fitting the energy spectrum of the effective Hamiltonian with that of the 
ab initio calculations, the parameters in the effective model can be determined. For GdBiTe 3 , our 
fitting leads to M = 27meV, A = 2.1eVA, B = 7.1eVA 2 , C = -UmeV, D = 4.9eVA 2 , 
A' PS 0, A e = l.leVA, g B M = ISmeV and g T M = 59meV, which agree well with the ab initio 
results. Such an effective model can be used for illustration of the formation of the GdBiTe 3 QAH 
system. As shown in Fig. 2c-e, when we have no FM ordering, the fact that M , B > suggests 
it is the topologically trivial insulator as SQL LaBiTe 3 , and the lack of inversion symmetry splits 
the double degeneracy except at T point. As we adiabatically increase the exchange field M, the 
\B+, \ |) and |T + , | 7) states start to move towards each other. At the transition point, those 
two bands form a 2D Dirac cone, and as we further increase the exchange field, those two bands 
become inverted and the Ak± term hybridizes them and creates a band gap again. In this process, 
a topological phase transition of SQL GdBiTe 3 is shown clearly from a topologically trivial phase 
to the QAH phase. 



Our theoretical calculations show that a SQL of GdBiTe 3 is a 2D stoichiometric magnetic 
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topological insulator, realizing the long-sought-after QAH state. This topological material can be 
grown by MBE, or by exfoliating the bulk crystal. In addition, YBiTe 3 with the same structure is 
found to be a normal insulator^, and could serve as the best substrate for MBE growth. Experi- 
mentally, the best way to see the QAH effect is to measure the four-terminal Hall conductance as 
a function of gate voltage. A quantized plateau in Hall conductance should be observed when the 
chemical potential is inside the gap. These intrinsic QAH materials could be used for applications 
with dissipationless electronic transport. 
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Figure 1: | Crystal Structure, Brillouin zone and band structure, a, Bulk crystal structure of 
GdBiTe 3 . A quintuple layer(QL) with Tel - Bi - Te2 - Gd - Tel' is indicated by the red box. 
b, Brillouin zone for GdBiTe 3 with space group R3m, which has four inequivalent time-reversal- 
invariant points T(0, 0, 0), L(7r, 0, 0),F(ir, n, 0) and Z(n, w, n). The projected(l 1 1) 2D Brillouin 
zone is marked by the blue hexagon with its high-symmetry k points T, K and M. c, d, e, Band 
structure with spin-orbit coupling(SOC) for (c) single QL Bi 2 Te3, (d) single QL LaBiTe3 and (e) 



single QL GdBiTe 3 . The Fermi level is fixed at OeV. 
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Figure 2: (Schematic representation of the topological phase transition. a,b, The phase 
transition from (a) the topologically trivial insulator phase of LaBiTe 3 to (b) the topologically 
non-trivial QAH phase of GdBiTe 3 . Because of the Kramers degeneracy, \p x — ip y , Bi + Tel, 7 
) and \p x + ip y , Bi + Tel, 1) states at the bottom of the conduction band, as well as states 
\Px + ipy, Tel' + Te2, 7) and \p x — ip y , Tel' + Te2, 1) at the top of the valence band are dou- 
ble degenerated for LaBiTe 3 . The time-reversal symmetry is broken for GdBiTe 3 because of 
the ferromagnetism of Gd with the half filled f bands. The Kramers degeneracy at T is re- 
moved. Due to the large SOC and the ferromagnetic moment, the band inversion occurs between 
\Px + iPy, Bi + Tel, |) and \p x + ip y , Tel' + Te2, 7). c,d,e, Phase transition based on the Low- 
energy effective model. The Kramers degeneracy at T for LaBiTe 3 is shown in the band structure 
(c) with the time-reversal symmetry, but the Ifr^mers degeneracy for the band structure of (e) 
GdBiTe 3 is broken due to the lack of the time-reversal symmetry. The gapless Dirac-type disper- 



sion is shown in (d) the band structure at the phase transition point. 
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ky kx 

Figure 3: | Phase diagram, a, The topological phase diagram depending on the lattice constant, 
do is GdBiTe 3 's experimental bulk lattice constant. The system is in topologically non-trivial QAH 
phase with the small lattice constant(a < 1.008a ) marked by yellow, and it becomes topologically 
trivial ferromagnetic insulator with the large lattice constant(a > 1.008a ) marked by blue, b, The 
topological phase diagram depending on the correlation U with fixed experimental lattice constant 
a . The GdBiTe 3 is in QAH phase with small U(< A.3eV), and it changes to the topologically 
trivial ferromagnetic insulator phase with large U(> 4.3eV). c, The 2D Dirac-type dispersion of 
the lowest conduction band and the highest valence band at the transition point. 
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Figure 4: |Edge states a,b, Energy and momentum dependence of the LDOS for GdBiTe 3 with 

(a) QAH phase, and with (b) topologically trivial ferromagnetic insulator phase. The red regions 

indicate the 2D bulk bands and the dark blue regions indicate the bulk energy gap. The edge state 

are clearly shown in the bulk energy gap. The edge states in (a), which connect the bulk conduction 

bands and the bulk valence bands, are chiral, and the detailed dispersions around T are zoomed in 

in the inset. Comparing with the case of QAH, (b) the topologically trivial ferromagnetic insulator 
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phase has no chiral edge states. The edge of the single QL is taken to be along x direction, which 
is the normal axis of the reflection symmetric plane, shown in the inset of (b). 
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